<html>
<head><meta charset="utf-8"><title>paged single-file serialization · t-compiler/wg-self-profile · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/index.html">t-compiler/wg-self-profile</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html">paged single-file serialization</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="208941816"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208941816" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208941816">(Sep 03 2020 at 09:31)</a>:</h4>
<p>Hey everyone, over the last couple of days I've looking into implementing a single-file serialization format roughly as described in <a href="https://github.com/rust-lang/measureme/issues/128">https://github.com/rust-lang/measureme/issues/128</a>.</p>



<a name="208941854"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208941854" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208941854">(Sep 03 2020 at 09:31)</a>:</h4>
<p>I have a working proof of concept implementation here: <a href="https://github.com/michaelwoerister/measureme/tree/paged_sinks">https://github.com/michaelwoerister/measureme/tree/paged_sinks</a></p>



<a name="208942132"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208942132" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208942132">(Sep 03 2020 at 09:34)</a>:</h4>
<p>it implements two variants: </p>
<ul>
<li>a simpler one that writes pages to disk on the thread that does the recording (<a href="https://github.com/michaelwoerister/measureme/blob/paged_sinks/measureme/src/paged_serialization_sink2.rs">https://github.com/michaelwoerister/measureme/blob/paged_sinks/measureme/src/paged_serialization_sink2.rs</a>), and </li>
<li>a more sophisticated one that lets a background thread do the writing and clearing of pages (<a href="https://github.com/michaelwoerister/measureme/blob/paged_sinks/measureme/src/paged_serialization_sink.rs">https://github.com/michaelwoerister/measureme/blob/paged_sinks/measureme/src/paged_serialization_sink.rs</a>).</li>
</ul>



<a name="208942207"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208942207" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208942207">(Sep 03 2020 at 09:35)</a>:</h4>
<p>The background thread implementation is up to 5% faster than the simpler implementation in the micro benchmarks. But it is also more complicated.</p>



<a name="208942296"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208942296" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208942296">(Sep 03 2020 at 09:36)</a>:</h4>
<p>I'm not sure how valid the micro benchmarks are for assessing real-world performance in the compiler.</p>



<a name="208942980"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208942980" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208942980">(Sep 03 2020 at 09:44)</a>:</h4>
<p>Overall I think the paging approach works pretty well.</p>



<a name="208943103"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208943103" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208943103">(Sep 03 2020 at 09:45)</a>:</h4>
<p>(Credit goes to <span class="user-mention" data-user-id="125250">@Wesley Wiser</span> for coming up with the idea)</p>



<a name="208951668"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208951668" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208951668">(Sep 03 2020 at 11:19)</a>:</h4>
<p>We can probably evaluate pretty well by running the perf suite and seeing how much time it takes - less or more, once we get to integration point.</p>
<p>I am a bit worried about a background thread when we eventually look at parallel rustc more - but I suspect ultimately we can tackle that bridge then. Did we consider buffering everything in memory and then writing at the end? IIRC on perf.rlo we never ended up with more than a few hundred megabytes of data</p>



<a name="208951788"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208951788" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208951788">(Sep 03 2020 at 11:20)</a>:</h4>
<p>I guess maybe a scheme that buffers say 5 million events or so and then does a small burst on the separate thread to record them feels like it might be a bit better</p>



<a name="208956233"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208956233" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208956233">(Sep 03 2020 at 12:05)</a>:</h4>
<p>Yes, I'm not a big fan of libraries spawning off background threads either. OTOH, this kind of background worker would spend most of its time blocked waiting for new worked, or blocked waiting for the data to make it to the disk...</p>



<a name="208956408"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208956408" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208956408">(Sep 03 2020 at 12:07)</a>:</h4>
<blockquote>
<p>Did we consider buffering everything in memory and then writing at the end?</p>
</blockquote>
<p>The <code>MmapSerializationSink</code> does this. It seems to perform pretty well and in multithreaded mode it blows all the other implementations out of the water because it does not need a <code>Mutex</code>.</p>



<a name="208956459"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208956459" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208956459">(Sep 03 2020 at 12:07)</a>:</h4>
<p>but that only works because it works under the assumption that it has an unlimited amount of memory available.</p>



<a name="208956526"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208956526" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208956526">(Sep 03 2020 at 12:08)</a>:</h4>
<p>I'm not sure if that is a good trade off</p>



<a name="208956793"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208956793" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208956793">(Sep 03 2020 at 12:10)</a>:</h4>
<blockquote>
<p>I guess maybe a scheme that buffers say 5 million events or so and then does a small burst on the separate thread to record them feels like it might be a bit better</p>
</blockquote>
<p>Yes, that is what the proof of concept impl does. Each page is X bytes (128KB seems to work fine) and once that's full, it gets sent to the background thread which immediately will write it to disk.</p>



<a name="208956932"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208956932" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208956932">(Sep 03 2020 at 12:12)</a>:</h4>
<p>it's not super complicated (all the threading and synchronization takes place within the one source file and is well-contained), but it's still quite a bit harder to reason about than the single-threaded version (especially when it comes to error modes)</p>



<a name="208957287"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208957287" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208957287">(Sep 03 2020 at 12:15)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> Regarding testing on perf.rlo: the postprocessing tools on the server obviously can't deal with the new file format. If I test a rustc with the new <code>measureme</code> version, will I still get the high-level timings or will it just crash?</p>



<a name="208957717"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208957717" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208957717">(Sep 03 2020 at 12:19)</a>:</h4>
<p>Hm I'm not sure</p>



<a name="208957792"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208957792" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208957792">(Sep 03 2020 at 12:20)</a>:</h4>
<p>It'll probably crash, but I can make it not do that without too much trouble</p>



<a name="208957822"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208957822" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208957822">(Sep 03 2020 at 12:20)</a>:</h4>
<p>We can also provide new post processing tools? Are they not yet ready?</p>



<a name="208957897"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208957897" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208957897">(Sep 03 2020 at 12:21)</a>:</h4>
<p>At least for perf server we basically have unlimited memory (iirc, 32GB) for this use case</p>



<a name="208957922"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208957922" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208957922">(Sep 03 2020 at 12:21)</a>:</h4>
<p>No rustc will use more than 10GB on our cases, and that's a high estimate I suspect</p>



<a name="208958102"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208958102" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208958102">(Sep 03 2020 at 12:23)</a>:</h4>
<p>But other than bumping up to more than 128KB - that feels quite low, we probably want to pick as high a number as possible, right?</p>



<a name="208958118"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208958118" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208958118">(Sep 03 2020 at 12:23)</a>:</h4>
<p>I suspect that the background thread will be fine</p>



<a name="208984260"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208984260" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208984260">(Sep 03 2020 at 15:24)</a>:</h4>
<p>Wow <span class="user-mention" data-user-id="124287">@mw</span> nice work! I'm going to take some time later today to review the code and see if I can offer any feedback.</p>



<a name="208985793"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208985793" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208985793">(Sep 03 2020 at 15:35)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/187831-t-compiler.2Fwg-self-profile/topic/paged.20single-file.20serialization/near/208958102">said</a>:</p>
<blockquote>
<p>But other than bumping up to more than 128KB - that feels quite low, we probably want to pick as high a number as possible, right?</p>
</blockquote>
<p>I got this idea from how a lot of SQL databases store their clustered indexes. I think many of those try to align with the default OS page size which is usually 4kb I believe. I don't see any compelling reason to go that small for our use case though. Larger page sizes should mean we "allocate" less often which should have lower overhead on average.</p>



<a name="208985879"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208985879" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208985879">(Sep 03 2020 at 15:35)</a>:</h4>
<blockquote>
<p>It'll probably crash, but I can make it not do that without too much trouble</p>
</blockquote>
<p>IIRC we had a hack at one point to run two different versions of the tooling to handle this case.</p>



<a name="208985990"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208985990" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208985990">(Sep 03 2020 at 15:36)</a>:</h4>
<p>Yeah, we still have that :)</p>



<a name="208986102"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986102" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986102">(Sep 03 2020 at 15:37)</a>:</h4>
<p>Is the "for real" fix to ship the tools in rustup? I think we talked about that at one point.</p>



<a name="208986105"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986105" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986105">(Sep 03 2020 at 15:37)</a>:</h4>
<p>in any case I'm not too worried about making the transition, we can manage it pretty well I suspect one way or another</p>



<a name="208986230"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986230" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986230">(Sep 03 2020 at 15:38)</a>:</h4>
<p>rustup would help for the summarize which runs synchronously -- but it doesn't help for the flamegraph and crox support recently added to perf, which is on-demand live</p>



<a name="208986294"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986294" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986294">(Sep 03 2020 at 15:38)</a>:</h4>
<p>But for that my suspicion is that we can just have perf.rlo frontend depend on multiple versions of the analyzeme libraries</p>



<a name="208986310"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986310" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986310">(Sep 03 2020 at 15:38)</a>:</h4>
<p>Ah ok</p>



<a name="208986377"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986377" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986377">(Sep 03 2020 at 15:39)</a>:</h4>
<p>We should have a meta tool that can read the various file headers and invoke the correct version of the tool for you</p>



<a name="208986423"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986423" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986423">(Sep 03 2020 at 15:39)</a>:</h4>
<p>Hm, bundling up all versions of the tooling into one? That would be great!</p>



<a name="208986455"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986455" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986455">(Sep 03 2020 at 15:39)</a>:</h4>
<p>I guess that may not work if you try to, for example, diff between two different versions</p>



<a name="208986465"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986465" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986465">(Sep 03 2020 at 15:39)</a>:</h4>
<p>well, I don't see why not</p>



<a name="208986477"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986477" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986477">(Sep 03 2020 at 15:39)</a>:</h4>
<p>if the two event streams are the same</p>



<a name="208986521"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986521" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986521">(Sep 03 2020 at 15:40)</a>:</h4>
<p>like, the event type is probably the same (or has the same API, anyway)</p>



<a name="208986579"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986579" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986579">(Sep 03 2020 at 15:40)</a>:</h4>
<p>Oh, we process to json right?</p>



<a name="208986604"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986604" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986604">(Sep 03 2020 at 15:40)</a>:</h4>
<p>well, for summarize, sure</p>



<a name="208986643"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986643" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986643">(Sep 03 2020 at 15:40)</a>:</h4>
<p>Then yeah that would mostly work then.</p>



<a name="208986650"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986650" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986650">(Sep 03 2020 at 15:40)</a>:</h4>
<p>but I mean that e.g. summarize wouldn't try to read the events file directly</p>



<a name="208986690"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986690" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986690">(Sep 03 2020 at 15:40)</a>:</h4>
<p>but it would instead call a "analyzeme-combined" crate that knows how to read any version of events files</p>



<a name="208986702"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986702" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986702">(Sep 03 2020 at 15:41)</a>:</h4>
<p>Oh I see</p>



<a name="208986736"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986736" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986736">(Sep 03 2020 at 15:41)</a>:</h4>
<p>so you can just directly call it with different versions and it basically just works</p>



<a name="208986747"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986747" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986747">(Sep 03 2020 at 15:41)</a>:</h4>
<p>I was thinking of something more along the lines of what rustup(?) does when you invoke <code>cargo</code> or <code>rustc</code></p>



<a name="208986799"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986799" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986799">(Sep 03 2020 at 15:41)</a>:</h4>
<p>That would certainly work though</p>



<a name="208986867"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986867" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986867">(Sep 03 2020 at 15:42)</a>:</h4>
<p>yeah, that's a rustup feature</p>



<a name="208986911"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986911" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986911">(Sep 03 2020 at 15:42)</a>:</h4>
<p>I think it works poorly for our use case because we <em>do</em> want to compare across, at least somewhat</p>



<a name="208986961"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986961" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986961">(Sep 03 2020 at 15:42)</a>:</h4>
<p>and I <em>think</em> it might actually be simpler with my proposal</p>



<a name="208986978"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208986978" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208986978">(Sep 03 2020 at 15:42)</a>:</h4>
<p>since you only need to have the "shared" logic in one place</p>



<a name="208987004"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208987004" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208987004">(Sep 03 2020 at 15:43)</a>:</h4>
<p>(and you don't have multiple binaries etc etc)</p>



<a name="208992215"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208992215" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208992215">(Sep 03 2020 at 16:16)</a>:</h4>
<p>At one point I wanted to just keep the old deserializers for each version around in the decoder. But having a "dispatcher" crate that just references old versions of <code>analyzeme</code> sounds pretty smart!</p>



<a name="208993220"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208993220" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208993220">(Sep 03 2020 at 16:23)</a>:</h4>
<p>Oh interesting... can we import old versions of the library from <a href="http://crates.io">crates.io</a> and then rename them?</p>



<a name="208993251"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208993251" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208993251">(Sep 03 2020 at 16:23)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/187831-t-compiler.2Fwg-self-profile/topic/paged.20single-file.20serialization/near/208958102">said</a>:</p>
<blockquote>
<p>But other than bumping up to more than 128KB - that feels quite low, we probably want to pick as high a number as possible, right?</p>
</blockquote>
<p>There's a tradeoff here: The bigger the pages the longer it takes to write it to disk, but also the fewer times we have to go through the flushing overhead. With the background thread approach it doesn't matter too much (I observed no difference at all between 128KB and 512KB for example). With a synchronously approach you have to find some middle ground between stalling often for short periods and less often for longer periods. In an interactive scenario (e.g. for computer games in a garbage collected language) you clearly want the former. In the case of <code>rustc</code>, I'm not so sure what's better. Ultimately, and luckily, it doesn't matter too much, I think <code>:)</code></p>



<a name="208993605"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208993605" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208993605">(Sep 03 2020 at 16:25)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/187831-t-compiler.2Fwg-self-profile/topic/paged.20single-file.20serialization/near/208957822">said</a>:</p>
<blockquote>
<p>We can also provide new post processing tools? Are they not yet ready?</p>
</blockquote>
<p>The tools work just fine, they don't need to change at all fortunately. But for doing a perf run of an unmerged PR they wouldn't be available on the server yet.</p>



<a name="208994013"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208994013" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208994013">(Sep 03 2020 at 16:27)</a>:</h4>
<blockquote>
<p>With a synchronously approach you have to find some middle ground between stalling often for short periods and less often for longer periods.</p>
</blockquote>
<p>We might want to represent that time in the profile somehow. To make it clear the time spent isn't from a parent event but from mm itself.</p>



<a name="208995471"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208995471" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208995471">(Sep 03 2020 at 16:37)</a>:</h4>
<p><span class="user-mention silent" data-user-id="124287">mw</span> <a href="#narrow/stream/187831-t-compiler.2Fwg-self-profile/topic/paged.20single-file.20serialization/near/208993605">said</a>:</p>
<blockquote>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/187831-t-compiler.2Fwg-self-profile/topic/paged.20single-file.20serialization/near/208957822">said</a>:</p>
<blockquote>
<p>We can also provide new post processing tools? Are they not yet ready?</p>
</blockquote>
<p>The tools work just fine, they don't need to change at all fortunately. But for doing a perf run of an unmerged PR they wouldn't be available on the server yet.</p>
</blockquote>
<p>It should be easy enough to have summarize be in place and we can create dummy string-index and string-data files so the upload stuff doesn't break</p>



<a name="208995508"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208995508" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208995508">(Sep 03 2020 at 16:37)</a>:</h4>
<p>I think async on a background thread that is mostly idle should be fine</p>



<a name="208995616"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/208995616" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#208995616">(Sep 03 2020 at 16:38)</a>:</h4>
<p>my thought with larger chunks was that in the "small crate" case the larger the chunk is the more likely we flush at the end of compilation, not interfering with cache lines etc at all</p>



<a name="209013552"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209013552" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209013552">(Sep 03 2020 at 18:55)</a>:</h4>
<p>I ran the micro benchmarks with a number of different page sizes. Note that there's quite a bit of variability in those numbers.</p>
<table>
<thead>
<tr>
<th>PAGE SIZE</th>
<th>ASYNC</th>
<th>SYNC</th>
</tr>
</thead>
<tbody>
<tr>
<td>4KB</td>
<td>5.953s</td>
<td>5.744s</td>
</tr>
<tr>
<td>64KB</td>
<td>5.216s</td>
<td>5.312s</td>
</tr>
<tr>
<td>128KB</td>
<td>5.076s</td>
<td>5.145s</td>
</tr>
<tr>
<td>512KB</td>
<td>5.038s</td>
<td>5.132s</td>
</tr>
<tr>
<td>1MB</td>
<td>5.157s</td>
<td>5.329s</td>
</tr>
<tr>
<td>4MB</td>
<td>5.231s</td>
<td>5.260s</td>
</tr>
<tr>
<td>10MB</td>
<td>4.944s</td>
<td>5.172s</td>
</tr>
<tr>
<td>16MB</td>
<td>5.204s</td>
<td>5.399s</td>
</tr>
<tr>
<td>64MB</td>
<td>5.432s</td>
<td>5.455s</td>
</tr>
<tr>
<td>512MB</td>
<td>7.440s</td>
<td>6.953s</td>
</tr>
</tbody>
</table>



<a name="209013953"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209013953" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209013953">(Sep 03 2020 at 18:58)</a>:</h4>
<p>Interesting. I guess this number can be toggled up and down relatively easily anyway, so doesn't matter much</p>



<a name="209014054"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209014054" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209014054">(Sep 03 2020 at 18:59)</a>:</h4>
<p>Yeah, it seems that the page size basically doesn't matter if it's somewhere between 100KB and 16MB</p>



<a name="209014249"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209014249" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209014249">(Sep 03 2020 at 19:00)</a>:</h4>
<p>I just ran the 10 MB test case again and it gives me 5.172s (ASYNC) and 5.391s (SYNC), so it's not really better than the other cases.</p>



<a name="209014457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209014457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209014457">(Sep 03 2020 at 19:02)</a>:</h4>
<p>Really interesting that the sync case is faster than the async case for 4kb pages. Which is probably what your OS pagesize is set to.</p>



<a name="209014511"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209014511" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209014511">(Sep 03 2020 at 19:02)</a>:</h4>
<p>Unless that's just noise.</p>



<a name="209014617"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209014617" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209014617">(Sep 03 2020 at 19:03)</a>:</h4>
<p>let me run that again. I'm pretty sure it's noise</p>



<a name="209015482"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209015482" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209015482">(Sep 03 2020 at 19:09)</a>:</h4>
<p>Here's the full output from <code>hyperfine</code>:</p>
<div class="codehilite"><pre><span></span><code>Benchmark #1: ./target/release/paged_sync_single_threaded
  Time (mean ± σ):      5.750 s ±  0.108 s    [User: 3.719 s, System: 1.989 s]
  Range (min … max):    5.626 s …  5.967 s    10 runs

Benchmark #2: ./target/release/paged_async_single_threaded
  Time (mean ± σ):      5.880 s ±  0.089 s    [User: 4.313 s, System: 2.556 s]
  Range (min … max):    5.787 s …  6.067 s    10 runs
</code></pre></div>


<p>The SYNC case is still faster. But I would not read too much into these numbers <code>:)</code></p>



<a name="209015630"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209015630" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209015630">(Sep 03 2020 at 19:10)</a>:</h4>
<p>(<code>hyperfine</code> is awesome!)</p>



<a name="209015707"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209015707" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209015707">(Sep 03 2020 at 19:11)</a>:</h4>
<p>I could believe there's some special magic that happens when our page size happens to be the same as the OS's, at least on Linux.</p>



<a name="209015742"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209015742" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209015742">(Sep 03 2020 at 19:11)</a>:</h4>
<p>But yeah, that doesn't seem compelling enough to use 4kb over 128kb or something.</p>



<a name="209015861"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209015861" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209015861">(Sep 03 2020 at 19:12)</a>:</h4>
<p>Unless the async version is so complex that we don't want to ship it</p>



<a name="209015873"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209015873" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209015873">(Sep 03 2020 at 19:12)</a>:</h4>
<p>(Haven't looked yet sorry!)</p>



<a name="209099615"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209099615" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209099615">(Sep 04 2020 at 14:27)</a>:</h4>
<p><span class="user-mention" data-user-id="125250">@Wesley Wiser</span> At this point I would not recommend to review in detail. I think the main question to answer now is whether to go with the background thread approach or with the in-thread approach. But actually, it would be even better to do some performance testing on perf.rlo with both approach. If the single-threaded approach turns out to be fast enough, then it would just go with that.</p>



<a name="209099774"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209099774" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209099774">(Sep 04 2020 at 14:28)</a>:</h4>
<p>I plan to do the performance testing on perf.rlo, btw</p>



<a name="209116402"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209116402" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209116402">(Sep 04 2020 at 16:51)</a>:</h4>
<p>Well the async version wasn't as complex as I thought it would be :)</p>



<a name="209116474"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209116474" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209116474">(Sep 04 2020 at 16:52)</a>:</h4>
<p>More complexity than the sync for sure but not an enormous amount.</p>



<a name="209116796"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209116796" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209116796">(Sep 04 2020 at 16:55)</a>:</h4>
<p>I do like the sync version more as I think it's quite a bit easier to understand. I'm also really interested in trying a version of that where we use TLS variables instead of the mutex so each thread writes to their own pages.</p>



<a name="209116923"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209116923" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209116923">(Sep 04 2020 at 16:56)</a>:</h4>
<p>I have some small nit-type suggestions for both of the approaches but nothing major <span aria-label="smile" class="emoji emoji-1f642" role="img" title="smile">:smile:</span></p>



<a name="209130496"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209130496" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209130496">(Sep 04 2020 at 19:08)</a>:</h4>
<p><span class="user-mention silent" data-user-id="125250">Wesley Wiser</span> <a href="#narrow/stream/187831-t-compiler.2Fwg-self-profile/topic/paged.20single-file.20serialization/near/209116402">said</a>:</p>
<blockquote>
<p>Well the async version wasn't as complex as I thought it would be :)</p>
</blockquote>
<p>Except that multi-threaded stuff is always 10x more complex than you think it is <code>:)</code></p>



<a name="209130828"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209130828" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209130828">(Sep 04 2020 at 19:11)</a>:</h4>
<p>With paging a TLS approach might actually be doable, yeah! This code spends a <em>lot</em> of time doing Mutex locking and unlocking. But with something like 128 KB pages I don't see a real resource problem with allocating a writer per page.</p>



<a name="209131006"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209131006" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209131006">(Sep 04 2020 at 19:13)</a>:</h4>
<p>I'm not sure if there are other problems. IIRC, values in TLS are properly dropped when a thread exits, so that should be fine.</p>



<a name="209131139"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209131139" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209131139">(Sep 04 2020 at 19:14)</a>:</h4>
<p><span class="user-mention" data-user-id="125250">@Wesley Wiser</span>, let me know about those nits, so I can work them in as I go.</p>



<a name="209131164"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209131164" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209131164">(Sep 04 2020 at 19:14)</a>:</h4>
<p>Oh sure! What's the best way for me to give them to you?</p>



<a name="209131174"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209131174" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209131174">(Sep 04 2020 at 19:14)</a>:</h4>
<p>Just write them up here?</p>



<a name="209131193"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209131193" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209131193">(Sep 04 2020 at 19:14)</a>:</h4>
<p>yes</p>



<a name="209131234"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209131234" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209131234">(Sep 04 2020 at 19:15)</a>:</h4>
<p>doesn't have to be now though :)</p>



<a name="209131370"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209131370" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209131370">(Sep 04 2020 at 19:16)</a>:</h4>
<p>I'm waiting on a build to finish so I might as well write down what's off the top of my head :)</p>



<a name="209134218"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209134218" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209134218">(Sep 04 2020 at 19:48)</a>:</h4>
<p>Sync code:</p>
<blockquote>
<p>assert!(num_bytes + PAGE_HEADER_SIZE &lt;= self.shared_state.page_size);</p>
</blockquote>
<p>This constraint should be documented somehow. We should not be recording ~127.9kb strings though so it shouldn't be an issue in practice. </p>
<p>Async code:</p>
<blockquote>
<div class="codehilite"><pre><span></span><code>                    // A zero-length page is the signal for stopping the
                   // background thread.
</code></pre></div>


</blockquote>
<p>Unless there's a noticeable performance difference, I think we should use an enum here <code>enum Msg { Write(Vec&lt;u8&gt;), Close  }</code>.</p>
<blockquote>
<p>drop(sx.send(payload));</p>
</blockquote>
<p>I thought this was to get a <code>Drop::drop()</code> call to run but I think this is just to get rid of the "unused result" warning? <code>let _ = sx.send(payload);</code> might be clearer.</p>
<p>Both:</p>
<blockquote>
<p>fn write_page_header</p>
</blockquote>
<p>I think it would be nice if this had an actual <code>struct</code> that got encoded as the header. I think we will want to have additional fields in the header eventually. (For instance, we may want to let the caller control the size of the page. We would then need to write that value into the header so we can set up the reader appropriately. Another example, we could encode the endianness in the header so that BE systems don't have to convert to LE during the recording.)</p>



<a name="209136594"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209136594" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209136594">(Sep 04 2020 at 20:16)</a>:</h4>
<blockquote>
<p>values in TLS are properly dropped when a thread exits</p>
</blockquote>
<p>I would hesitate to make this a unilateral claim, at least not on all platforms</p>



<a name="209136607"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209136607" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209136607">(Sep 04 2020 at 20:16)</a>:</h4>
<p>e.g. I think today on macOS the main thread won't do this? Not sure though</p>



<a name="209166539"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209166539" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209166539">(Sep 05 2020 at 07:43)</a>:</h4>
<p>Ah yes, the docs say that none of the Unix systems do this. Something to keep in mind.</p>



<a name="209312447"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209312447" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209312447">(Sep 07 2020 at 16:36)</a>:</h4>
<p><span class="user-mention" data-user-id="124287">@mw</span> should I build and fallback to the self-profile tooling built from your branch?</p>



<a name="209312455"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209312455" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209312455">(Sep 07 2020 at 16:36)</a>:</h4>
<p>(Is it already producing just one file?)</p>



<a name="209312531"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209312531" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209312531">(Sep 07 2020 at 16:37)</a>:</h4>
<p>looks like yes</p>



<a name="209607006"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209607006" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209607006">(Sep 10 2020 at 08:06)</a>:</h4>
<p>OK, we got some numbers from <a href="https://github.com/rust-lang/rust/pull/76436">https://github.com/rust-lang/rust/pull/76436</a>. It looks like the background-thread version is not worth the hassle -- which is good news <code>:)</code></p>



<a name="209607046"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209607046" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209607046">(Sep 10 2020 at 08:07)</a>:</h4>
<p>I'd like to do a clean re-implementation of the whole thing now.</p>



<a name="209607455"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209607455" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209607455">(Sep 10 2020 at 08:12)</a>:</h4>
<p>I think it makes sense to get rid of <code>SerializationSink</code> pluggability altogether in the process. I think usage so far has shown we just use one or the other based on the platform, but that they perform pretty much the same. We can still have platform specific implementations if necessary but that would only be an internal implementation detail of  <code>measureme</code>, not something the is exposed to clients of the library.</p>



<a name="209607540"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209607540" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209607540">(Sep 10 2020 at 08:13)</a>:</h4>
<p>Although it's interesting that some platforms seem to use the <code>ByteVecSerializationSink</code>. I wonder what's that about</p>



<a name="209607977"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209607977" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209607977">(Sep 10 2020 at 08:18)</a>:</h4>
<blockquote>
<p>Although it's interesting that some platforms seem to use the <code>ByteVecSerializationSink</code>. I wonder what's that about</p>
</blockquote>
<p>OK, that seems to be better solved by just disabling the profiler in cases where the platform doesn't allow writing to files at all.</p>



<a name="209608048"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209608048" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209608048">(Sep 10 2020 at 08:19)</a>:</h4>
<p>What do you folks, think: Shall we remove <code>SerializationSink</code> from the public interface of <code>measureme</code> altogether?</p>



<a name="209641601"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209641601" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209641601">(Sep 10 2020 at 13:42)</a>:</h4>
<p><span class="user-mention silent" data-user-id="124287">mw</span> <a href="#narrow/stream/187831-t-compiler.2Fwg-self-profile/topic/paged.20single-file.20serialization/near/209608048">said</a>:</p>
<blockquote>
<p>What do you folks, think: Shall we remove <code>SerializationSink</code> from the public interface of <code>measureme</code> altogether?</p>
</blockquote>
<p>This seems overall like a good idea to me. I've gotten feedback from 3rd parties trying to use <code>measureme</code> that there's a lot of "internal" details that bleed through the api and makes it a lot harder to get started with. I think simplifying the api where we can is good.</p>



<a name="209752462"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209752462" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209752462">(Sep 11 2020 at 08:19)</a>:</h4>
<p>Yes, I'd guess the whole string handling story is the hardest part.</p>



<a name="209752596"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/209752596" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#209752596">(Sep 11 2020 at 08:20)</a>:</h4>
<p>I'll go ahead and prepare some PRs then <code>:)</code></p>



<a name="211027746"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/211027746" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#211027746">(Sep 23 2020 at 16:26)</a>:</h4>
<p>I think the info in this article explains why the version with the background thread is only marginally faster than the one persisting things synchronously: <a href="https://lwn.net/Articles/457667/">https://lwn.net/Articles/457667/</a></p>



<a name="211027914"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/211027914" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#211027914">(Sep 23 2020 at 16:27)</a>:</h4>
<p>That is, on Linux at least, writing data to a file means that it is actually only copied to some memory that is managed by the kernel which then does the writing asynchronously in the background.</p>



<a name="211031951"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/211031951" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#211031951">(Sep 23 2020 at 17:00)</a>:</h4>
<p>There's also all kinds other ways of writing data to disk. E.g. setting <code>O_DIRECT</code> when opening a file on Linux, one can skip those kernel buffers and write to disk directly. In that case the memory being written needs to be OS page aligned. That might be related to what you've seen in databases, <span class="user-mention" data-user-id="125250">@Wesley Wiser</span>.</p>



<a name="211032229"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/211032229" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#211032229">(Sep 23 2020 at 17:02)</a>:</h4>
<p>But I think the bottleneck for <code>measureme</code> right now is the mutex when writing to the buffer, so I won't spend too much time on optimizing the other parts.</p>



<a name="211039644"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/211039644" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#211039644">(Sep 23 2020 at 17:58)</a>:</h4>
<p><span class="user-mention silent" data-user-id="124287">mw</span> <a href="#narrow/stream/187831-t-compiler.2Fwg-self-profile/topic/paged.20single-file.20serialization/near/211032229">said</a>:</p>
<blockquote>
<p>But I think the bottleneck for <code>measureme</code> right now is the mutex when writing to the buffer, so I won't spend too much time on optimizing the other parts.</p>
</blockquote>
<p>Sorry for the random idea, but I think file appends are atomic, at least under a certain limit. That might make it possible to omit acquiring a mutex. Here's an interesting SO comment about this: <a href="https://stackoverflow.com/a/35256561">https://stackoverflow.com/a/35256561</a></p>



<a name="211242829"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/211242829" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#211242829">(Sep 25 2020 at 10:52)</a>:</h4>
<p>That's interesting!</p>



<a name="211246888"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/211246888" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#211246888">(Sep 25 2020 at 11:43)</a>:</h4>
<p>OK, <a href="https://github.com/rust-lang/measureme/pull/132">https://github.com/rust-lang/measureme/pull/132</a> is ready for review.</p>



<a name="211246978"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/211246978" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#211246978">(Sep 25 2020 at 11:44)</a>:</h4>
<p>^ <span class="user-mention" data-user-id="125250">@Wesley Wiser</span></p>



<a name="214267236"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/214267236" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#214267236">(Oct 23 2020 at 02:06)</a>:</h4>
<p>o/</p>



<a name="214267260"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/214267260" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#214267260">(Oct 23 2020 at 02:07)</a>:</h4>
<p>this is really cool! maybe <code>wg-self-profile</code> is where I should've kept all the <code>rdpmc</code> discussion too. one thing I didn't consider much is introducing breaking changes, had I realized there was this breaking change to batch it with</p>



<a name="214267269"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/214267269" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#214267269">(Oct 23 2020 at 02:07)</a>:</h4>
<p>anyway I came in here to check if there's <a href="https://github.com/rust-lang/rust/issues/77398">#77398</a> discussion but I guess not</p>



<a name="214268591"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/214268591" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#214268591">(Oct 23 2020 at 02:46)</a>:</h4>
<p>I think it's not a problem to bump readily if we need to</p>



<a name="216674745"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/paged%20single-file%20serialization/near/216674745" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/paged.20single-file.20serialization.html#216674745">(Nov 13 2020 at 21:22)</a>:</h4>
<p><span class="user-mention" data-user-id="125250">@Wesley Wiser</span> <span class="user-mention" data-user-id="124287">@mw</span> uhh just realized the docs still talk about 3 files, that's a bit unfortunate, since the README links to <a href="http://docs.rs/measureme">docs.rs/measureme</a></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>